@oceantologist you need to uninstall the ps3controller driver.

@mfreshwater

Thanks for your response.

You're welcome

And I just read your second link and that looks like exactly what I need to know. I hope that all the buttons correlate with the games like they should but that helps a ton! I will give those a shot tonight.

Also there are some Video Tutorials further down on that same document that might be of some help as well.

You can see which emulator you are using from the Runcommand screen (which is what pops up for a couple seconds when you first launch a game). If you have more than one emulator for a system install, you can change the emulator from the Runcommand. Here are the documents about Runcommand:
https://retropie.org.uk/docs/Runcommand/

Thanks for your answer, I tried other emulators but unfortunately, none of the games I want to play are emulated correctly so I want to stick with mupen64plus.

Am I really the only one who tried to implement different controller layouts with this emulator? Can't be true ...

@bassholio which n64 emulator?

@Furluge yeah i was the reason that raphael learned about the issue which affected a crap ton of controllers. Then he so awesomely found the bug in linux and created the fix for it. But yeah its been fixed for a long ole time. 😊

2 years later... thank you so much. I would have never figured that one out.

Had the same problem with retroarch emulators. Solved it by using joystick selection:

https://github.com/meleu/RetroPie-joystick-selection/blob/master/README.md

In mupen standalone you can define controllers via their device name, so they won't switch around.

@silentq Have you seen the new N64 controller by 8bitdo?
Here

Solved!
Nvm, I figured it out after reading the wiki for the 15th time. You need to disable hotkeys in the autoconf.cfg file first and then edit the mupen64plus.cfg to enable whatever you want to exit the emulator. As stated above I use Z Trigger + Start button to exit back out. It works perfectly and you do not sacrifice any buttons.

/opt/retropie/configs/all/autoconf.cfg

mupen64plus_hotkeys enable hotkey auto configuration (0)

then go in and edit

/opt/retropie/configs/n64/mupen64plus.cfg

and set your control to the way you want it to exit out of game. I programmed mine for Z trigger and Start hit together